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FORMAT CONVERTER USING BI-DIRECTIONAL MOTION VECTOR AND 
METHOD THEREOF 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

The present invention generally relates to an image signal format converter and an 

associated method of performing the conversion. More particularly, the invention relates to a 
5 format converter, and method, that performs frame rate conversion and de-interlacing using a 
bi-directional motion vector. The present application is based on Korean Patent Application 
No. 00-32388 filed on June 13, 2000, which is incorporated herein by reference. 

2. Description of the Related Art 

Generally, in a PC or a high definition TV (HDTV), conversion of a format such as a 

10 frame rate and/or de-interlacing is needed to exchange signals utilizing various signal 
standards. 

FIG. 1 is a block diagram of a conventional frame rate conversion apparatus. 

Referring to FIG. 1, an image dividing unit 1 10 divides an image into a changed 
region and an unchanged region for efficient motion estimation as shown in FIG. 2. The 
15 unchanged region is divided again into a covered region, an uncovered region, a background, 
and an object. 

A motion estimating unit 120 generates the motion vector of a block using a block 
matching algorithm which is generally known in the art of video coding. In a representative 
conventional block matching algorithm, a motion vector is determined for each block based 
20 on the assumption that pixels in a predetermined-size block, as shown in FIG. 3, moved 



without rotation, magnification, or reduction. In FIG. 3, it is assumed that the motion vector 
of an N X N size base block located on arbitrary coordinates (xc, yc) in the current frame (Q is 
estimated in a range of±P pixels from the previous frame (fp). Then, the search range in the 
previous frame is (N+2P) x (N+2P). Therefore, the motion vector is determined at a location 
5 which has the maximum correlation among the total (2P+1)^ candidate locations. 

At this time, the difference between the base block in the current frame and the 
candidate block in the previous frame is calculated as a mean absolute difference (MAD) by 
the following equation 1 : 

MAD = -^i;i;|fe + i,y, + j)- fp(x, +i + m,y,+j + n\ (1) 
■1^ 1=1 j=\ 

1 0 Here, the motion vector of the final block is determined at a location (m, n) of a 

search range for which the mean absolute difference between the current block and the 
candidate block is a minimum. 

A spatiotemporal smoothing imit 130 refines an inappropriate motion vector obtained 
in the motion estimating unit 120, and refines the smoothness of a motion vector as shown in 
15 FIG. 4. 

A motion-compensated interpolation unit 140 finds a forward motion vector for the 
previous frame and the next frame of an image to be interpolated and, using the obtained 
motion vector, performs bi-directional interpolation according to region classification 
information generated in the image dividing unit 110. At this time, as shown in FIG. 5, the 
20 motion-compensated interpolation using a forward motion vector generates overlaps, where 
blocks are overlapped because two or more motion vectors are assigned in a frame to be 
interpolated, and holes where a motion vector is not assigned. These overlaps and holes 
directly affect the picture quality of an image by causing a degradation in the picture quality. 



Also, since these overlaps and holes have irregular shapes, they should be handled on a pixel- 
by-pixel basis. Therefore, for the conventional apparatus, complicated signal processing 
processes and complicated hardware are necessary to remove the overlaps and holes. 

Also, for an ordinary television video signal, a frequency band is compressed using an 
interlacing method in which two fields form a firame. Recently, in a PC or an HDTV, for 
which the displays generally use a progressive scanning method to display an interlaced 
image, empty image lines, due to de-interlacing, are generated by an arbitrary method for 
progressive scanning. 

FIG. 6 is a conceptual basic diagram of ordinary de-interlacing. 

Referring to FIG. 6, de-interlacing changes a field containing only an odd-numbered 
pixel, or only an even-numbered pixel, into a frame. At this time, an output frame Fo( X , n) is 
defined by the following equation 2: 



Here, x means a spatial location, and n is a field number. F( x , n) is an input field, 
and Fi( x , n) is a pixel to be interpolated. 

FIG. 7 is a 3 X 3 window for applying an edge-based line averaging (ELA) de- 
interlacing algorithm which does not use motion compensation. 

Referring to FIG. 7, ELA de-interlacing uses the correlation between pixels 
considering direction (x, y) from the location of a pixel to be interpolated as shown in 
equation 3, below. That is, the mean value of pixels neighboring a pixel to be interpolated 
and pixels to be interpolated in the previous field and the next field of the field to be 
interpolated is output, as follows: 




{y mod 2 = n moi 
otherwise 




(2) 



f(x-u^ -Uy,n)+F(x + u^ +Uy,n)/2, if min(a,b, c) = a 
Fo(x,n) = - f(x-u^ +Uy,n)+F(x + u^ -Uy,n)/2, if min(a,b,c) = b 

f(x - u y , n) + f(x + u y , n)/ 2, otherwise 
here,a = |f(x-u^ -Uy,n)-F(x + u^ +Uy,n]j 
b = |F(x-ii^ +iiy,n)-F(x + u^ '^y'^]! 
c = |f(x - ii y , n)- f(x + Uy , n| 



(3) 



FIG. 8 is a conceptual diagram for explaining an ordinary time-recursive (TR) de- 
interlacing method. 

Referring to FIG. 8, TR de-interlacing using a motion vector assumes that the 
5 previous field (n-1) is perfectly de-interlaced, and the missing data of the current field (n) is 
compensated with a motion vector. A pixel to be interpolated can be the original pixel of the 
previous field, or a previously interpolated pixel. Therefore, a pixel to be interpolated can be 
expressed as the following equation 4: 

, s \Fi^,n), {y modi = n modi], 

[F{x - d\x, n),n-l), otherwise 

10 However, since the ELA de-interlacing method does not use motion compensation, 

flickering occurs in regions where movement exists and, since the TR de-interlacing method 
is continuously de-interlaced, an error that occurs in an arbitrary field can be propagated to 
other fields. 

SUMMARY OF THE INVENTION 

15 To solve the above problems, it is an object of the present invention to provide a 

frame rate converting method which refines a picture quality by dkectly obtaining a bi- 
directional motion vector of two continuous frames for a pixel to be interpolated. 
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It is another object to provide a frame rate converter using the frame rate converting 
method. 

It is another object to provide a de-interlacing method wherein by estimating a bi- 
directional motion vector between two continuous fields for a pixel to be interpolated, the 
5 method is easy to perform and has an excellent outline-keeping ability. 

It is another object to provide a de-interlacing apparatus using the de-interlacing 
method. 

It is another object to provide a de-interlacing apparatus which can enhance reliability 
=5 of motion information and reduce errors in a pixel to be interpolated by adaptively selecting 
^10 between motion-compensated interpolation value and spatiotemporal interpolation value 
according to the degree of motion in an input image. 

1 To accomplish the above-mentioned objects of the present invention, there is provided 

■2 a frame rate converting method having the steps of (a) estimating a bi-directional motion 

vector for a frame to be interpolated using motion vectors between the current frame and the 
rs previous frame; (b) setting the motion vector of a neighboring block, having the minimum 

error distortion among motion vectors estimated in step (a), in a frame to be interpolated, as 

the motion vector of the current block; and (c) forming a frame to be interpolated with the 

motion vector set in step (b). 

To accomplish another object of the present invention, there is also provided a frame 
20 rate converter having a bi-directional motion estimating means for obtaining the motion 
vector between the current frame and the previous frame, assigning the motion vector to a 
frame to be interpolated, and estimating the assigned motion vector for a frame to be 
interpolated; a spatiotemporal smoothing unit for evaluating the accuracy of the motion 
vector of the current block in the frame to be interpolated in the bi-directional motion 



estimating means, and then setting the motion vector of a neighboring block, which has the 
minimum error distortion, as the motion vector of the current block; and an interpolation unit 
for extending the block to be interpolated, and interpolating with the motion vector obtained 
in the spatiotemporal smoothing unit in an overlapped region with different weights. 

5 To accomplish another object of the present invention, there is also provided a de- 

interlacing method including (a) estimating a bi-directional motion vector for a pixel to be 
interpolated using motion vectors between the previous field and the next field; (b) setting the 
motion vector for which neighboring error distortion is the minimum in step (a) as the motion 
vector of a pixel to be interpolated; and (c) forming the pixel to be interpolated with the 
}0 motion vector set in step (b). 

To accomplish another object of the present invention, there is also provided a de- 
interlacing apparatus having a bi-directional motion estimating means for obtaining the 
motion vector between the current field and the previous field, assigning the motion vector to 
a field to be interpolated, and estimating the assigned motion vector for a field to be 

15 interpolated; a spatiotemporal smoothing unit for evaluating the accuracy of the motion 
vector of the current block in the field to be interpolated in the bi-directional motion 
estimating means, and then setting the motion vector of a neighboring block, which has the 
minimum error distortion, as the motion vector of the current block; and a signal converting 
unit for forming a pixel of a line having no data, with the median value of pixel values 

20 obtained by applying the motion vector set in the spatiotemporal smoothing unit, the mean 
value of the pixel values, and the values of pixels vertically neighboring the pixel to be 
interpolated. 



BRIEF DESCRIPTION OF THE DRAWTNOS 

The above objects and advantages of the present invention will become more apparent 
by describing in detail a preferred embodiment thereof with reference to the attached 
drawings in which: 

FIG. 1 is a block diagram of a conventional frame rate conversion apparatus; 

FIG. 2 is a diagram for explaining an image dividing method in the image dividing 
unit of FIG. 1; 

FIG. 3 is a diagram for explaining a motion estimating method in the motion 
estimating unit of FIG. 1; 

FIG. 4 illustrates a screen before refinement and a screen after refinement in the 
spatiotemporal smoothing unit of FIG. 1; 

FIG. 5 is an example of the structure of an image interpolated by motion 
compensation in the motion-compensated interpolation unit of FIG. 1; 

FIG. 6 is a conceptual basic diagram of ordinary de-interlacing; 

FIG. 7 is a 3x3 window for applying edge-based Hne averaging (ELA) de-interlacing 
algorithm which does not use motion compensation; 

FIG. 8 is a diagram for explaining motion vector estimation for each block in the 
spatiotemporal smoothing imit of FIG. 6; 

FIG. 9 is a block diagram of a frame rate conversion apparatus according to the 
present invention; 

FIGS. lOA through IOC are conceptiial diagrams for obtaining a bi-directional motion 

vector; 



FIG. 1 1 is a conceptual diagram for refining a motion vector of the spatiotemporal 
smoothing unit of FIG. 9; 

FIG. 12 is a conceptual diagram for explaining a motion-compensated interpolation 
method of the refined motion-compensated interpolation unit of FIG. 9; 

FIG. 13 is a block diagram of a de-interlacing apparatus according to the present 
invention; 

FIG. 14 is a conceptual diagram for showing decimation conversion of the motion 
estimating unit of FIG. 13; 

FIG. 15 is a conceptual diagram for showing motion-compensated de-interlacing in 
the signal conversion unit of FIG. 13; 

FIG. 16 is a conceptual diagram for showing spatiotemporal interpolation using a 
median filter in the signal conversion unit of FIG. 13; and 

FIG. 17 is another embodiment of a de-interlacing apparatus according to the present 
invention. 

DETAILED DESCRIPTION OF THE TNVRNTTON 

Hereinafter, embodiments of the present invention will be described in detail with 
reference to the attached drawings. The present invention is not restricted to the following 
embodiments, and many variations are possible within the spirit and scope of the present 
invention. The embodiments of the present invention are provided in order to more 
completely explain the present invention to anyone skilled in the art. 

FIG. 9 is a block diagram of a frame rate conversion apparatus according to the 
present invention 



The apparatus of FIG. 9 includes a motion estimating unit 210, a spatiotemporal 
smoothing unit 220, and a refined motion-compensated interpolation unit 230. 

Referring to FIG. 9, the motion estimating unit 210 obtains a motion vector between 
the current frame and the previous frame, and assigns the motion vector to a frame to be 
interpolated, and estimates a bi-directional motion vector for the frame to be interpolated. 

The spatiotemporal smoothing unit 220 evaluates the accuracy of the motion vector of 
the current block estimated for the frame to be interpolated, and then sets a motion vector of 
neighboring blocks, for which error distortion is the minimum, as the motion vector of the 
ciurent block. 

The refined motion-compensated interpolation unit 230 forms a block to be 
interpolated with the mean of blocks in the previous frame and the next frame of the frame to 
be interpolated, using the motion vector obtained in the spatiotemporal smoothing unit 220. 
At this time, the refined motion-compensation interpolation unit 230 extends the block to be 
interpolated and interpolates with different weights in an overlap region. 

FIGS. lOA through IOC are conceptual diagrams for obtaining a bi-directional motion 

vector. 

First, in neighboring frames, Fn-i is the (n-l)-th frame, Fn+i is the (n+l)-th frame, and 
F„ is the n-th frame. In the n-th frame (F^), as shown in FIGS. lOA through IOC, a bi- 
directional motion vector is obtained through a motion vector initialization stage (FIGS. lOA 
and lOB) and a motion vector adjusting stage (FIG. IOC). 

Referring to FIG. lOA, the motion vector initialization stage will now be explained. 
First, (n-l)-th frame/field (Fn.i) and (n+l)-th frame/field (Fn+i) are decimated in 2:1 
ratio, and reconstructed to (n-l)-th frame/field (f„_j ) and (n+l)-th frame/field (f„^i ). 



Then, as shown in FIG. lOA, the (n+l)-th frame/field (f^^^j ) is divided into a plurality 
of blocks, and a search range in ) for each block is determined. Then, within the search 
range, by applying a block matching algorithm (BMA) to (f^_i ) and (f„^i ), a forward motion 
vector (MV) is estimated for each block. Then, as shown in FIG. lOB, the n-th frame/field 
5 (f„ ) to be interpolated is also divided into blocks, and the estimated forward MV for each 
block is set as an initial MV for each block of the n-th frame/field (f^^ ) to be interpolated. 

Then, by compensating for motion according to the block grid for each block using a 
bi-directional motion vector as shown in FIG. 1 OB, overlaps and holes do not occur in the 
existing video signal. That is, as shown in FIG. lOB, the forward motion vector is initialized 
10 (by simply moving the motion vector into a block) to match with the block grid of the n-th 

frame/field (f^ ) to be interpolated. Accordingly, overlaps and holes are not generated similar 
to the way they are generated when a forward motion vector is estimated based on the (n-1)- 
th frame/field (f„_i ). This is because the block grid and the interpolated blocks are matched. 

Next, referring to FIG. IOC, the motion vector adjusting stage will now be explained. 

1 5 First, since the forward MV is used in the initialization stage, without the benefit of a bi- 
directional motion vector, the initial MV, for the field to be interpolated, is not precise. To 
account for this imprecision, taking the forward MV obtained in the motion vector 
initialization stage as an initial value, a small search range (+ d) is newly set in (f„_j ) and 
(f„^j ) . Then, using a BMA again in the small search range (± d), matching blocks in 

20 frame/field (f„ ) to blocks in frames/fields (f„_j ) and (f„^i ), the motion vector set in the 
initial stage is corrected, and a bi-directional motion vector is generated. 
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To explain the adjusting stage for the initial MV shown in FIG. IOC, an arbitrary 
block (Bti) in the n-th trame/field (f^ ) to be interpolated is considered. The center of the 
block (Bti) is located at coordinates (x,y), and corresponds to the initial MV ((d. v)). At 
this time, the initial MV {d. ) simultaneously shows the motion between the n-th frame/field 
(f„ ) to be interpolated for the arbitrary block (Bti) and the (n+l)-th frame/field (f„^, ) as well 
as the motion from the (n-l)-th frame/field (f„_j ) to the n-th frame/field (f„ ) to be 
interpolated. Then, if the arbitrary block (Bti) in the n-th frame/field (f„ ) to be interpolated 
were to move in accordance with the initial MV {d. ), the arbitrary block (Bti) would be 
generated by the block (Bti) of the (n-l)-th frame/field (f„_i ) and the block (Ba) of the 
(n+l)-th frame/field (f^^i ). That is, the centers of the initial block (Bti) and block (Bt2) can 
be expressed by the following equation 5: 

^t2ixn^yt2)={x>y)+(h,v)=(x + h,y + v) 

Here, the arbitrary block (Bti) is located on a fixed location, and each of block (Bti) 
and block (Bt2) moves from the initial location within the search range (+ d). At this time, if 
the n-th frame/field (f, ) should be located in the center between the (n-l)-th frame/field 
{K-i ) and the (n+l)-th frame/field the motion between the block (Bti) and the 

arbitrary block (Bti) should be the same as the motion between the arbitrary block (Bti) and 
the block (Bo). For this, on the motion trajectory of the initial MV, the block (Bti) and the 
block (Bt2) should move symmetrically from the center of the block (B^) to be interpolated. 

Therefore, the number of possible combinations when the search range is ± is (2d + 

if. Subsequently, the bi-directional vector between the (n-l)-th frame/field (f„_, ) and the 
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(n+l)-th frame/field [f^^^ )for the n-th frame/field [f„ ) is obtained. At this time, if the n-th 
frame/field [f^ ) should be at the center between the (n-l)-th frame/field (f„_j ) and the (n+1)- 
th frame/field [f„_^, ), the motion vector to each direction would have the same value. 

FIG. 11 is a conceptual diagram for refining a motion vector of the spatiotemporal 
smoothing unit 220 of FIG. 9. 

Referring to FIG. 1 1, first, in the frame/field to be interpolated, the current block 
under consideration is designated, MVq, candidate MV blocks surrounding the current block 
are designated MVi (i=l,. . .,8), and the motion vector of the current block is designated!) . 
Then, the motion vector of a block having the minimum displaced frame difference (DFD) 
among the motion vectors of the candidate blocks surrounding the current block is set as the 
motion vector of the current block. That is, in accordance with equation 6 below, using the 
bi-directional motion vector between two neighboring frame/fields, the DFD of the current 
block is obtained and the motion vector of the candidate block having the minimmn DFD is 
set as the motion vector of the current block. As a result, the spatiotemporal smoothing 
process refines picture quality by removing inaccurate motion vectors detected in the motion 
estimation. 

DFD(D)= Y.K{p--D)-f,,{p + D} (6) 

In equation 6, B(p) represents the interpolated block,/? represents the position of a 
pixel within a block, and D represents the motion vector of a block. 

FIG. 12 is a conceptual diagram for explaining a motion-compensated interpolation 
method of the refined motion-compensated interpolation unit 230 of FIG. 9 



12 



Referring to FIG. 12, the refined motion-compensated interpolation unit 230 forms a 
frame to be interpolated after taking the block mean of the two neighboring frames according 
to equation 7, below, using the motion vector obtained bi-directionally. At this time, the 
frame to be interpolated extends the original block size horizontally and vertically, and is 
5 interpolated in the overlapped region with different weights. 

^M = \Kip-D{B{p)))-f,,{p-D{Bm (7) 

FIG. 13 is a block diagram of a de-interlacing apparatus according to the present 
invention. 

Referring to FIG. 13, Fn-i, which is input first, is the (n-l)-th field, Fn is n-th field, and 
10 Fn+i is (n+l)-th field. F„ is a video signal which is converted from the n-th field (Fn) for 
progressive scanning. 

A motion estimating unit 410 obtains the motion vector of the n-th field (Fn), which 
corresponds to the location of the field to be interpolated, after obtaining the bi-directional 
motion vector, from the (n-l)-th field (F^.i) and the (n+l)-th field (Fn+i). The bi-directional 
1 5 motion vector to be obtained in the n-th field (Fn) is obtained by calculating the fields, for 
which decimation conversion has been performed, through the motion vector initialization 
stage (FIGS. lOA and lOB) and the motion vector adjusting stage (FIG. IOC). As a result, for 
the field to be interpolated, the bi-directional motion vector between the previous field and 
the next field is calculated. 

20 A spatiotemporal smoothing unit 420, as illustrated in FIG. 1 1, obtains a bi- 

directional motion vector, which is smoothed through spatiotemporal smoothing, because bi- 
directional motion vectors obtained in the motion estimating unit 410 have some 
discontinuities. 



A signal converting unit 430, which is an interlaced-to-progressive conversion block, 
restores any "no-data" lines in the n-th field (Fn) with the mean of pixels to which bi- 
directional motion vectors generated in the spatiotemporal smoothing unit 420 are applied, 
and outputs the fmal frame F„ . 

FIG. 14 is a conceptual diagram for showing decimation conversion of the motion 
estimating unit 410 of FIG. 13. 

Referring to FIG. 14, the (n-l)-th field (Fn-i) and the (n+l)-th field (F„+i) are input and 
reconstructed to the (n-l)-th field and the (n+l)-th field F„^^ , only with the lines having 
data. That is, the reconstructed (n-l)-th field and (n+l)-th field F^^j are reduced from 
the input (n-l)-th field (Fn-i) and (n+l)-th field (Fn+O by half, vertically. Therefore, the 
reconstructed (n-l)-th field F„_, and (n+l)-th field are decimated in a 2:1 ratio vertically 
and horizontally. 

FIG. 15 is a conceptual diagram for showing motion-compensated de-interlacing in 
the signal conversion unit 430 of FIG. 13. 

Referring to FIG. 15, "no-data" lines of the n-th field (Fn) are restored, using the bi- 
directional motion vector of the field (f„ ) to be interpolated. The restoration process can be 
expressed by the following equation 8: 

P _ F„_, {x-h,y-v) + F„,X^ + h,y + v) 

r„ ^ («) 

Here, x and y are the abscissa value and the ordinate value, respectively, in each field, 
and h and v are the horizontal component and the vertical component, respectively, of the bi- 
directional motion vector. 
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FIG. 16 is a conceptual diagram for showing spatiotemporal interpolation using a 
median filter in the signal conversion unit of FIG. 13. 

The performance of a de-interlacing method is affected greatly by the result of motion 
estimation. Therefore, to reduce any error in motion estimation, "no data" lines in the field 

to be interpolated are interpolated using a median filter as shown in FIG. 16. This 
interpolation can be expressed by the following equation 9: 

(F„ {p), if p is in the existing line, 

Median^A,B,QD,^^^^^, otherwise 

Here, pixels A, B, C, and D are defined as follows: 

A = F„^-Uy),B = F„^ + Uy),C = F,.j^-D]iD = F„,,(p + D) 

Here, D is a bi-directional motion vector, is (0, 1)^, and (C+D)/2 is the resulting 
value of the motion-compensated de-interlacing according to equation 9. 

Accordingly, if the median filter is used, the frame (/; ) to be finally output takes the 
value of the original pixel if the line has data, and otherwise is interpolated with the median 
value of the pixel (C) of the (n-l)-th field, the pixel (D) of the (n+l)-th field, pixels (A, B) 
which are vertically neighboring the pixel (Z) to be interpolated in the n-th field, and de- 
interlaced pixel ((C+D)/2), as the pixel (Z) of the n-th field. 

FIG. 17 is another embodiment of a de-interlacing apparatus according to the present 
invention. 

Referring to FIG. 17, a motion-compensated interpolation unit 172 interpolates with 

the mean of pixels using the interpolation value of a firame, that is a motion vector, as shown 

in FIG. 13 according to the present invention, or outputs the median value of pixel values, to 
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which a motion vector is appHed, the mean value of the pixels, and the value between two 
pixels which are vertically neighboring a pixel to be interpolated. 

A spatiotemporal interpolation unit 176 outputs the mean value of pixels neighboring 
a pixel to be interpolated and pixels on the locations to be interpolated in the previous field 
5 and the next field of the field to be interpolated, as the interpolation value of a frame. 

A motion evaluation unit 174 evaluates the degree of motion using the MAD value of 
the current block calculated in the motion estimating unit 410 of FIG. 13. 

A motion adaptation unit 178 sets the value of a pixel to be finally interpolated by 
adaptively calculating the output value of the motion-compensated interpolation unit 172 and 
1 0 the output value of the spatiotemporal interpolation unit 176 according to the degree of 
motion evaluated in the motion evaluation imit 174. 

Therefore, the de-interlacing apparatus of FIG. 17 prevents the error which occurs 
when an inaccurate motion vector is used in the process for determining the presence of 
motion. 

1 5 As described above, according to the present invention, by obtaining the bi-directional 

motion vector between two frames for a frame to be interpolated, overlaps and holes do not 
occur. Therefore, picture quality can be improved and particularly, panning or zooming 
images created by moving the camera can be efficiently processed. Also, noise on a time axis 
between fields, and flickers between lines, which occur in the conventional methods, can be 

20 reduced, and the ability to keep outhnes is better than the existing de-interlacing methods. 
Also, by adaptively selecting between a motion-compensated interpolation value and a 
spatiotemporal interpolation value according to the degree of the motion of an input image, 
reliability of information regarding motion is enhanced as compared to the method which 
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simply uses a motion-compensated interpolation value, and artifacts can be efficiently 
reduced. 
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